my thoughts on the xz backdoor
2024-03-31 by: flushy@flushy.net
From: flushy@flushy.net ------------------------------------------------------ gist version of this is here: https://gist.github.com/gonoph/c41630716d594e61a69477760ac045ae This is written in Markdown format. # XZ Backdoor summary and thoughts. TL;DR: This is terrifying. Basically, someone infiltrated the xz upstream, then started committing code that injected a backdoor into the xz library via the test suite via special crafted rogue compressed test files and obfuscated shell code. The end result is that sshd when patched to work with systemd (to notify systemd it's ready) would link to liblzma and execute the backdoor upon login attempts. # Longer version: The technical aptitude is overshadowed by the [social engineering and breech of trust][1] or [bribe of blackmail of a legitimate user (AV-601)][2]. [1]: https://attack.mitre.org/techniques/T1199/ [2]: https://sap.github.io/risk-explorer-for-software-supply-chains/#/attacktree?av=AV-601 So some of the evidence for this is "locked up" as GitHub suspended both maintainers of xz and the project. There is a mirror of xz on the project website, but at this point "[everything is suspect][3]" until proven otherwise. [3]: https://news.ycombinator.com/item?id=39870098 There is also a [FAQ put together by Sam James][4], and a [response by the other maintainer, Lasse Collin][5]. [4]: https://gist.github.com/thesamesam/223949d5a074ebc3dce9ee78baad9e27 [5]: https://tukaani.org/xz-backdoor/ ## Background [Andres Freund posted his story on oss-security][6]. Basically he noticed on his debian system that ssh server had some odd timing issues after upgrading lzma/xz-utils. He started poking around and found some odd tracing behavior, and noticed some odd calls being made to libraries: (a) WHEN the calls where made - aka sshd should not be making those calls to that library at that time, (b) WHERE in the library the calls were happening, and (c) it was consuming a lot of cpu inside the library and inside sshd. [6]: https://www.openwall.com/lists/oss-security/2024/03/29/4 After some investigation, he learned that there was a backdoor implanted in liblzma.so at BUILD TIME, and crucially it was implanted in (a) PLAIN SIGHT, (b) in FOUR PHASES, and (c) via one of the PROJECT ADMINS. ### Implanted in PLAIN SIGHT Jia Tan (@JiaT75) committed changes to xz and released two versions of xz and xz-utils - one in Jan of 2024, and the other in Feb of 2024. Then as is common, he crafted the corresponding "tarball" archives to go with the releases - and signed them with his private GPG key. @JiaT75 also [committed to several other projects][7], and started a campaign to encourage other maintainers of other distros to use the new released version of xz - but highlighted to use the archive and not the github source. [7]: https://gist.github.com/thesamesam/223949d5a074ebc3dce9ee78baad9e27#other-projects Phase 1: Starting in January (at least what we know about THIS BACKDOOR - there could be more), the author, Jia Tan (@JiaT75), committed several changes under the guise of adding compressed files (blobs) to be used in the test harness of the lzma library. This is very common, and not unusual for something like a compression library. Phase 2: In February, @JiaT75 started the next phase: backdoor activation. The test files uploaded to the source repository in January were actually obfuscated shell and C code. To activate the backdoor, Jia added some fixes to the test harness, resolved some real issues, fixed some code that would cause the backdoor to not work in some environments, and then made another release. Crucially, the tarball that was in the release did not have the same code as the git repository - which is NOT NORMAL. This tarball contained the activation code to inject the backdoor into the source at build time. If you installed from the git source, then this activation was not present. Phase 3: Jan and someone else then embarked on a campaign to influence other distro maintainers to (a) upgrade to the newest release of xz, (b) to only use the release archive (tarball), and (c) to only open any bugs or concerns with him [directly instead of publicly][8]. Jan did this to Fedora, but [someone else][9] attempted to do this for Debian. [8]: https://news.ycombinator.com/item?id=39866275 [9]: https://news.ycombinator.com/item?id=39868390 Phase 4: Jan and possibly others in parallel efforts to Phase 1, 2, and 3 - had submitted PR requests to various projects that could have potentially detected the backdoor. One being [xz to disable sandboxing][10] (notice the lone period "+." in CMakeLists - it would fail the compile test). Another being [oss-fuzz][11], where it looks like the author helped the maintainer of oss-fuzz unknowingly disable a check that was preventing the liblzma exploit from working in this test environment. The question being: "this PR could have prevented discovery of this issue?" Also looks like they attempted to release new versions for [MingGW, arch linux, and alpine][12]. [10]: https://git.tukaani.org/?p=xz.git;a=commitdiff;h=328c52da8a2bbb81307644efdb58db2c422d9ba7 [11]: https://github.com/google/oss-fuzz/pull/10667 [12]: https://news.ycombinator.com/item?id=39870098 ## Technical workings ### Payload extraction When using the release tarball, the backdoor was injected into the code via the build process. The author obfuscated the code inside a compressed test file, which was extracted, uncompressed and passed to a shell to be ran directly. Another payload was also extracted from parts of the test file, de-mangled from "bad compress data" into "good compress data" and then uncompressed and saved as a payload source to be compiled along with the original source. This payload was only extracted and built on: 1) x86_64 architectures 2) and Debian or RPM based systems 3) detection if the distro uses systemd ### Payload execution Bulk taken from the [oss-security thread][6] and [Sam James][4] above. sshd utilizes systemd-notify to notify systemd that it is ready for connections. So sshd pulls in libsystemd which pulls in liblzma. When the liblzma library is 1st loaded, it performs checks to see if it should be enabled: 1) not running in a terminal (TERM is blank) 2) it's executed as /usr/sbin/sshd 3) LD_DEBUG and LD_PROFILE are blank 4) LANG is set 5) not running in a debugger (mostly works) If these pass, then the payload scans symbol tables in memory, adds an audit hook to the dynamic linker to scan additional symbols that haven't loaded, and once it detects certain function symbols from the sshd executable, then it injects a hook for those functions to run it's own payload 1st. This is implemented as a "store and forward" aka: saving the old value, updating the old function pointer to itself, and calling the saved function value when it's payload completes. Once it has injected itself as a store and forward, any incoming ssh connection to the sshd server is intercepted via the payload during the authorization phase of the connection, and the best guess at this point is that the payload is designed to let a specific set of SSH private keys (or fingerprints of them) to log into the system by bypassing the normal authorization process. One thing that Sam James makes clear that I agree with: > We know that the combination of systemd and patched openssh are > vulnerable > but pending further analysis of the payload, we cannot be certain that > other > configurations aren't. ## Opinions and Aftermath This did not affect any production versions of any Linux distros, but this type of attack is terrifying. ### Broken Supply Chain This makes me question everything about the supply chain. It will also add a bigger burden to folks like Red Hat (Disclaimer: I work at Red Hat) and other distributions. They will need to perform more extensive due diligence for any upstream code incorporated into their distributions. In the end, this did not affect any production versions of any distros, but it raises some questions. * "How did this happen?" * "How can we prevent this?" * "How can we detect this?" It also raises a question about * "Do we know who the maintainers are?" @JiaT75 used his private GPG key to sign commits and the release binaries. Has anyone met him? Can we be sure that he is who he says he is? What level of trust do the folks that know him and signed his key: do they trust that the holder of that private key is him? Do others that signed the signers keys trust that those folks are who they say they are? Rinse and Repeat. Trust networks are complicated things and very difficult to built organically, especially in this era of remote work and faceless communication. ### How did this happen? Lasse Collin has been the sole maintainer of xz-utils for 15 years. In 2022,[he mentioned][13] he was [struggling][14] to keep up with demand and had long term mental struggles. I understand this far too well. Was this an "attack of opportunity" or was it "state sponsored?" I'm not sure anyone knows, yet. The complexity of the attack, coupled with the long term infiltration, the targeted architectures, VPNs used, and the [patch timelines][15] - it suggests multiple people in multiple timezones. [13]: https://www.mail-archive.com/xz-devel@tukaani.org/msg00567.html [14]: https://news.ycombinator.com/item?id=39873552 [15]: https://twitter.com/birchb0y/status/1773871381890924872 ### How can we prevent this? I'm not really sure how to prevent this. There are two main issues: 1) the technical problems 2) the people problems #### Technical problems At minimum upstreams should be running heuristics on code and identify anomalies. Things like [oss-fuzz][16] is a great start. Also Mitre goes through several scenarios and mitigation techniques for those scenarios. Examples relevant being [Trusted Supply Chain][17] and [Trusted Relationship][18] attacks. Also GPG key [trusts and signing][19] and [building the web of trust][20] and [here][21]. In some circles, we've grown lax around this process, and we shouldn't. [16]: https://github.com/google/oss-fuzz [17]: https://attack.mitre.org/techniques/T1195/ [18]: https://attack.mitre.org/techniques/T1199/ [19]: https://www.gnupg.org/gph/en/manual/x334.html [20]: https://www.gnupg.org/gph/en/manual/x547.html [21]: https://www.phildev.net/pgp/gpgtrust.html #### The People Problems Folks get burned out. They have trauma. They have debt. They have relationship problems. They fall madly in love and drop everything else. Their pets pass away. They lose a parent or they suddenly find they need to care for one. They have a baby or start a family. They can pass silently in their sleep - like a friend of mine, leaving behind a legacy of an open source karaoke project. Point is: humans are complex and life can be fragile. Lots of things can change the motivation and passion for something into a burden or into something that just feels insignificant. We should contribute to projects: either time and money, but what about those projects that are ran by the lone-wolf? I don't want corporate entities to sudden take interest in these projects and take them over, but at the same time, we need stronger communities in these spaces. We should be identifying these at risk projects, and then finding ways to help. This incident brings to mind the [XKCD comic][22]. [22]: https://xkcd.com/2347/ # DISCLAIMER I'm a Red Hat employee, but this post/document is my own personal words, feelings, and research into this matter. It does not represent Red Hat, nor does it represent any of my colleagues or associates. --b